Ik gebruik de wiki_dump_reader om de data om te zetten naar tekstbestanden. Uiteraard heb ik weer de inmiddels gebruikelijke uitdagingen om de tekst om te zetten van unicode naar ascii. Waarom is dat toch zo lastig? Zelfs de speciale modeule 'unicodedata' die ik hiervoor installeer blijkt nog steeds fouten op te leveren bij het wegschrijven naar een tekstbestand.
Dus maar weer de 'harde methode met een karakteromzettting met de beulpzame module 're'.
from wiki_dump_reader import Cleaner, iterate
import unicodedata
import re
file1 = open(r'C:\\Users\\DirkW\\Documents\\AJW2\\' + "WikiTextFile.txt",mode='w',encoding='utf-8')
# unicode to ascii converter
def unicode_to_ascii(s):
return ''.join(c for c in unicodedata.normalize('NFD', s)
if unicodedata.category(c) != 'Mn')
cleaner = Cleaner()
def preprocess_sentence(w):
w = unicode_to_ascii(w.strip())
#w = unicode_to_ascii(w.lower().strip())
# creating a space between a word and the punctuation following it
# eg: "he is a boy." => "he is a boy ."
# Reference:- https://stackoverflow.com/questions/3645931/python-padding-punctuation-with-white-spaces-keeping-punctuation
w = re.sub(r"([?.!,¿])", r" \1 ", w)
w = re.sub(r'[" "]+', " ", w)
# replacing everything with space except (a-z, A-Z, ".", "?", "!", ",")
w = re.sub(r"[^a-zA-Z?.!,¿]+", " ", w)
w = w.rstrip().strip()
'''
while w.find('. .')!= -1:
w = w.replace('. .','')
while w.find(' ')!= -1:
w = w.replace(' ',' ')
# adding a start and an end token to the sentence
# so that the model know when to start and stop predicting.
'''
w = w + ' <|endoftext|> '
return w
#for title, text in iterate('C:\\Users\\DirkW\\Documents\\AJW2\\nlwiki-20191001-pages-articles-multistream1.XML-P1P123351'):
i = 0
for title, text in iterate('C:\\Users\\DirkW\\Documents\\AJW2\\nlwiki-20191001-pages-articles-multistream.XML'):
text = cleaner.clean_text(text)
cleaned_text, _ = cleaner.build_links(text)
file1 = open(r'C:\\Users\\DirkW\\Documents\\AJW2\\Wiki\\' + "WikiTextFile" + str(i) + ".txt",mode='w',encoding='utf-8')
file1.write(preprocess_sentence(cleaned_text))
file1.close
i+=1
Ik maak hiermee allemaal losse wiki bestandjes van. 211.674 stuks (hij stopt met een error vanwege ruite gebrek. vermoedelijk maximaal aantal bestanden per folder?)
De teksten staan op 1 regel en worden afgesloten met '<|endoftext|>'
Ik laat het gpt-2 model een ruime 54.000 keer trainen. Hierbij gaat de gemiddelde 'loss' van rond de 3 (bij 10.000 epochs) naar rond de 2.50. Het lijkt nu niet heel veel beter meer te worden.
[10000 | 4285.37] loss=2.90 avg=2.97
[10001 | 4285.69] loss=3.12 avg=2.97
[10002 | 4286.01] loss=2.80 avg=2.97
[10003 | 4286.33] loss=2.97 avg=2.97
[10004 | 4286.64] loss=2.92 avg=2.97
[10005 | 4286.96] loss=3.42 avg=2.97
[10006 | 4287.27] loss=3.14 avg=2.97
[10007 | 4287.58] loss=3.24 avg=2.97
[52647 | 7822.63] loss=2.61 avg=2.51
[52648 | 7822.94] loss=3.07 avg=2.52
[52649 | 7823.24] loss=2.93 avg=2.52
[52650 | 7823.55] loss=2.29 avg=2.52
[52651 | 7823.86] loss=2.62 avg=2.52
Dan maar eens testen. Grappig genoeg lijkt hij Engels nog steeds best wel aardig te doen:
Model prompt >>> . i was trying to get it set up this morning and am running into an issue
======================================== SAMPLE 1 ========================================
on my pc Delete .... then i disable all the test emails you placedcase on adobe and then
======================================== SAMPLE 2 ========================================
when that happens i have this remote desktop connected straight from the command line rand resync sql excel <
======================================== SAMPLE 3 ========================================
because the dev team is unable to show all the fields . how can i turn off the settings ,
======================================== SAMPLE 4 ========================================
with a couple things . the install description lists it as south american and i would like to see
======================================== SAMPLE 5 ========================================
with cryptow can we all let in update to make it work ok ? when should i start m
================================================================================
Of
Model prompt >>> i was trying to get it set up this morning and am running into an issue
======================================== SAMPLE 1 ========================================
. do you know what this issue was ? med venlig hilsen best regards linesker rep
======================================== SAMPLE 2 ========================================
. i can read the see below . it is very unusual to receive bugs in files .sometimes called
======================================== SAMPLE 3 ========================================
with it or perhaps not met a few migrations ? and ooln , dit is epona
======================================== SAMPLE 4 ========================================
at the moment . it is still not working very well and i will either move to the next step
======================================== SAMPLE 5 ========================================
are those issue resolved ? other workday gigs are expected on b th and declare tomorrow ? case closed
================================================================================
Nederlands lijkt echter nog behoorlijk ver weg.
Model prompt >>> . beste marjan , heb je dit destijds nog kunnen testen ? ik vermoed dat het een ticket is die nog openstaat maar
======================================== SAMPLE 1 ========================================
die van betekenis komt voor . kun jij het scher
======================================== SAMPLE 2 ========================================
maandag enige tijd augustus beste marjan , is het mog
======================================== SAMPLE 3 ========================================
wellicht ook op jouw permissies in de website zoekt zit zo
======================================== SAMPLE 4 ========================================
op deze link moeten worden . dan heeft nog zelf en k
======================================== SAMPLE 5 ========================================
hij ziet dat een dekke kost voor een ticket is . groet
================================================================================
of
Model prompt >>> . beste marjan , heb je dit destijds nog kunnen testen ? ik vermoed dat het een ticket is die nog openstaat maar
======================================== SAMPLE 1 ========================================
ze het vertelde hij zo goed gedaan met eventueel nog kg .
======================================== SAMPLE 2 ========================================
opgesplitst voor de storing van de request wordt uiteindelijk ge
======================================== SAMPLE 3 ========================================
erkend is . anders kan ik neer een overeenkomst he
======================================== SAMPLE 4 ========================================
daar dus nog the ontvanger eine verplicht handigert over qu
======================================== SAMPLE 5 ========================================
maatregelen zichtbaar rondjamin zit . met vriendelij
================================================================================
Wellicht toch nog wat doortrainen of een keer van scratch proberen. Even een nachtje over slapen ...
