So, in my last project video, seen here:
I started a GPT2 124 project, following a Raff K. project, and have worked on it for some time now. This is the link:
https://github.com/nicholaskomsa/NetworkLib/tree/master/NetworkLibTest
Raff K project can be seen here:
GPT-2 from Scratch in C (Day 2/2)
Right now, ChatGPT2 forwards and produced output prediction. Seems really amazing, except there seems to be a problem — the output begins to repeat very quickly.
The tail of the first 1024 tokens of the test text, looks like so:
First Citizen:
Care for us! True, indeed! They ne’er cared for us
yet: suffer us to famish, and their store-houses
crammed with grain; make edicts for usury, to
support usurers; repeal daily any wholesome act
established against the rich, and provide more
piercing statutes daily, to chain up and restrain
the poor. If the wars eat us not up, they will; and
there’s all the love they bear
Concerning First Citizen, here is what ChatGPT2 124 proceeds to write:
for us, and the poverty they have to endure.
First Citizen:
We are not to be taken for fools, but for the people.
Second Citizen:
We are not to be taken for fools, but for the people.
MENENIUS:
What do you mean, what do you mean?
[repeats from First Citizen point for a continuous sequence of 3]
It seems to quickly repeat given input data, and the question is: error or not? It sure seems strange, to repeat…
I am working to understand GPT2 more right now and additionally working to understand Raff K code better, to help identify the source of the error if there is one. There is also a project by karpathy that I am thinking to reference before long.
I kind of feel like there could be a major problem with what I’ve written and it is quite concerning! However, the project persists…
Spot C++ 