ConlangDictionary 0.3 - now phonology parsing is faster

[faiuwle]

You downloaded them since January 20, right?

[Ċeaddawīc]

Yes, I just downloaded them today.

[faiuwle]

I can't reproduce it... what Windows are you running? You might try just running it again, or installing Qt and trying it again (similar to the Mac issue earlier).

[faiuwle]

So, like the new title says, I Aten't Dead. Here's the deal:

I was adding Features. One of the ideas was to have phonotactics specified using said features, making phonotactics easier for everyone. Unfortunately, it makes the orthography parsing harder for me. Also, it resulted in a whole lot of horrifying spaghetticode because of how things are stored in memory. Another thing I was planning to do was to actually hook it up to a real honest-to-god database, which should make everything faster and more efficient and generally much better, and since the backend was looking uglier and uglier every day, I decided to rewrite it entirely from scratch and recycle just the interface code. That's what I'm doing now. The database-ified version has compiled for the first time; it's like the dawn of a new era. I will be adding all the existing features back into it, and writing the new code to convert the existing XML saves and the new orthography parsers, but I don't expect a lot of difficulty, just a lot of code.

Anyway, the point is that it will take a bit a time, but it will be Awesome. Stay tuned.

ETA: Also, I will not be making any further modifications to the existing version.

[faiuwle]

I have been working on this - I've finished the phoneme-management and feature-management parts! (This leaves: word-management, phonotactics GUIs, and orthography-parsing algorithms, so by some measure I am like 2/5 done with the overhaul.)

Have some screenshots:
Main phoneme page (yes, that's an honest-to-god File menu):

Dialog for editing features (you can make features require each other, so +/-round requires labial):

Dialog for editing natural classes (the "Add/Edit Features" button on the main page launches just the right half of this dialog, for assigning features to phonemes individually);


Words will actually use the exact same feature system (saves on coding, too!); you can define "feature groups" like, say, "Part of Speech" that have subfeatures of (e.g.) "Verb", "Noun", "Adjective"... or you could borrow the system from phonology, and define +/-nominal, +/-verbal, +/-adjectival, and then define your gerunds as [+nominal][+verbal] or something. For the feature groups, you can assign different display options - so if you have a "Verb" group with subfeatures "Transitive" and "Intransitive", you can opt to have then displayed as "Transitive Verb" instead of "Verb: Transitive".

Because of the feature/natural class system, you'll be able to define your phonotactics as something like "Stop/Fricative + Nasal/Liquid + Glide" or analogous.

[CaesarVincens]

Awesome! Thanks for the update.

[Risla]

Yay, I've been looking forward to this.

I don't suppose you've included anything like alpha notation in the feature thing? It'd be really nice to specify something like [+nasal, αPLACE][-continuant, αPLACE] in the phonotactics to specify that nasals must be homorganic with any following stops. I have no idea how hard that would be to code in if you don't have it already, though.

[faiuwle]

I feel like I could probably work out the coding for that - it'd just be a matter of keeping track of the variables during the parsing, which will probably have to happen a bit anyway, since a simple state machine won't cut it. The part I'm not sure about is how to fit that information into the database. Would you ever want to do something like have [αPLACE] in one position and [LABIAL]/[DORSAL]/whatever in another? It would probably be fairly simple to set it up to say "all values for [some particular feature] in this sequence must match" and then track that at parsetime.

(Also, I've got the notation wrong, haven't I? It should be [feature, feature, feature] and not [feature][feature]. It's been a while.)

[Risla]

I feel like I could probably work out the coding for that - it'd just be a matter of keeping track of the variables during the parsing, which will probably have to happen a bit anyway, since a simple state machine won't cut it. The part I'm not sure about is how to fit that information into the database. Would you ever want to do something like have [αPLACE] in one position and [LABIAL]/[DORSAL]/whatever in another? It would probably be fairly simple to set it up to say "all values for [some particular feature] in this sequence must match" and then track that at parsetime.

(Also, I've got the notation wrong, haven't I? It should be [feature, feature, feature] and not [feature][feature]. It's been a while.)

That would be fantastic. I probably would want to have the ability to have [αPLACE] in one position and a specific place feature elsewhere; it seems like I do have such phonotactic constraints in some of my 'langs, but I can't think of any right now.

And now you have the notation right.

[alice]

This looks really neat.

Again, I'm sure that if all of us coders put our heads and creations together we could come up with a single and generally-usable piece of software which would meet the needs of just about any conlanger.

[faiuwle]

I probably would want to have the ability to have [αPLACE] in one position and a specific place feature elsewhere; it seems like I do have such phonotactic constraints in some of my 'langs, but I can't think of any right now.

That would be the hard thing; I'm not sure how to record that in the database, and the input would get tricky, too. I'll probably implement something slightly less powerful, and then release a prototype of sorts that lets you define features and phonotactics, and try to get an idea of what ideas people have/what else they would like out of the interface. (The reason why programmers hate GUIs: you ultimately spend a lot of time trying to guess what people actually want to do with it.)

This looks really neat.

Again, I'm sure that if all of us coders put our heads and creations together we could come up with a single and generally-usable piece of software which would meet the needs of just about any conlanger.

Heh, we always say that. I can post my schema, so far (although it looks like I'll be adding at least one more table for the phonotactics); it's SQLite, but if someone modified it to MySQL or something for their own purposes, I could modify the program to be able to access it as MySQL from a remote host, etc.

[Torco]

this is good. I am glad this exists. looking forward to moar

[faiuwle]

Ok, so I'm completely redesigning the phonotactics interface, and the parsing algorithms and everything, etc., and I once I get the basic interface done and hooked up to the database, I'd like to upload a kind of intermediate version with just the data entry, to gauge how effective the new interface and database design is going to be for people, at which point I can get the backend algorithms in gear, and we'll eventually have an actual version 0.3. I'm almost done with this, but getting hung up a bit on the phonotactics interface, mostly because I can't decide what would be most useful here, so I need some feedback before releasing the intermediate version. I could probably just whack something together and then rewrite it based on which parts of it everyone hates, but that seems kind of pointless.

Here's what the interface will do: you'll be able to define features, and group all your phonemes into natural classes (like "Consonant", or "Fricative") based on what features they have, as mentioned earlier. For the phonotactics, you'll be able to combine these classes to construct sequences like "Stop + Fricative|Nasal|Lateral + Glide" and specify that these sequences are legal onsets/codas/peaks. This might involve alpha-value features (if I can work it into the interface and the database) too. It's a complicated process, so it's easy to overcomplicate the GUI. How should this work?

These are the ideas I've been playing with, but I'm not really 100% happy with any of them:
1. The interface I was playing with before the overhaul - you build up a sequence by clicking things in the main tab, and then add the resulting sequence to the appropriate list (onsets/peaks/codas). It probably wouldn't be possible to edit/change things that are already in the lists, but it would be possible to delete them entirely and build another sequence to replace them. It's nice and simple, but it might not give you the control you'd want.
1. a. Maybe it could be possible to edit sequences individually with a dialog, but this seems like kind of a pain in the ass to me.
2. The tab shows mostly non-interactive lists of sequences, but there are buttons which launch dialogs where you can edit the whole list of possible onsets/peaks/codas all together. This consolidates things a bit, but that dialog could get way out of control in terms of interface complexity. OTOH, you get more direct control over the lists.
3. Rewrite it to be more like the other tabs, with the three master lists and an area off to the side (or the bottom) where you can edit individual properties of the selected list item, and the ability to add a new empty item or delete an existing one. To be honest, I am pretty sick of writing these tabs by now, but if that's more useful, it's not really a problem. The main issue is that these items are now more complicated list structures rather than a collection of scalar values, so it would probably be a lot more awkward, and annoying to hook up to the database.

I kind of want to avoid situations where you can add more GUI widgets to a layout dynamically (e.g. like in the word-phonology-editing dialog in the existing version) because it's likely to involve some layout-management related crashes at some point (as those dialogs did, if you recall), and lots of messy pointer juggling on my end. So the sequence being edited would probably be displayed as a text string and then modified by choosing things from drop-downs and pushing buttons and so forth. This will be regardless of the overall interface design.

Do any of those sound particularly useful/non-useful/painful/etc. from a user perspective? Do you have an idea for an interface for this, something that would be useful and sort of rational-looking? You can draw me pictures of the Ideal User Interface too, if you want.

[faiuwle]

Massive BUMP.

New version is out - please read the first post for details!

Man, porting this to Windows was so much grief you can't even imagine.

[Chagen]

Man, I want to love this, as it seems to have a bunch of really cool features (are you ever thinking about putting in an inflection engine?) but the UI for this is a mess, or my brain is fried today.

One thing I would really like is the ability to paste a string of IPA in, with spaces, and then define what that string is for the phonology. So for instance I could just put in /p b t d k g/ and then assign that all to +stop +consonant -vowel, or maybe /p t k/ and give that +consonant +stop -voice and then put in /b d g/ and give that +consonant +stop -voice. Right now you can only input each one in at a time.

I feel that I'll get used to the UI more if I try it out, but I'm really stuck on some things. For instance I wanted to try defining Sunbyaku's phonotactics, which are just (C)(v, y)V(F), but I can't seem to get the program to actually parse words through it. I put in the made-up word kuisarī [ku.i.sa.4i:] and while all of those phonemes are in and I put the full phonotactics in (along with the classes and features so it knows what to do with them), it can't seem to parse that word to IPA. Unless it's not supposed to do that, I dunno. Some documentation would be nice.

I'm not saying the program is bad, however, as it's a very strong piece of work. Just...a little confusing, I guess? I don't want to sound like I'm complaining.

[faiuwle]

You should be able to batch-add phonemes by putting spaces between them (e.g. "p t k" in the "add phoneme" box will add three separate phonemes) and then you can batch-assign phonemes to a certain natural class (which will add all the relevant features to them) by selecting multiple phonemes at a time with click+ctrl. It looks like I took out the line that automatically parses a word when you add it (oops), but if you click "update phonology" it should work.

It probably does need some documentation. A long time ago Torco was going to work on that, but that's probably not a thing anymore. I'll try to find time to write something up eventually.

[Chagen]

Oh you can add multiple ones at the same time? I didn't know that, and I don't mean to be rude, but that's where documentation can certainly help.

I must have messed something up, as Update Phonology does nothing. I'll look over my stuff and probably begin the dictionary file a new, I was just trying to figure out how things work so it's an unorganized mess right now

[faiuwle]

Weird, that works for me. Can I see your file?

EDIT: I updated the executables so that adding words now set the phonology as well.

EDIT2: I've written up some documentation.

[faiuwle]

Alright, so I was able to install an earlier Windows on VirtualBox, and produced a 32-bit Windows version instead of a 64-bit one.

If anyone is still interested in using this now...

EDIT: I think I found the bug that Chagan encountered, and fixed it. I'll update tomorrow.

[faiuwle]

Alright, update. Apparently my test dictionaries were ordered in such a way that this didn't happen, and I've mostly been playing with imported dictionaries than created new ones. Anyway, parsing word phonologies should work now.

3/17/14
- Adding a phoneme now also adds a default spelling.
- Word tab is now alphabetical.
- Fixed bug with word tab not updating things correctly.
- Fixed bug with editing features not resulting in parses afterwards.

[CaesarVincens]

Cool. I'll take a look. I was having some issue with the last version and the dictionary part proper, so I'll let you know if those are resolved.

Edit:

Well, it does work nicely now adding a phonological parse when I add the word, and I realize that my earlier trouble was from adding words that didn't conform to phonotactics (root forms).

[CaesarVincens]

I am having some trouble importing a lexical set from a txt file right now (it doesn't work). And export the lexicon to a txt file works, but also exports formatting code.

[faiuwle]

What does the file you're trying to import from look like?

Sorry about the formatting code in the export - it was doing that on some of my test dictionaries, but I thought it was an artifact from the old version getting translated over. I'll see if I can fix it.

EDIT: Did you import your dictionary from XML?

[CaesarVincens]

What does the file you're trying to import from look like?

Sorry about the formatting code in the export - it was doing that on some of my test dictionaries, but I thought it was an artifact from the old version getting translated over. I'll see if I can fix it.

EDIT: Did you import your dictionary from XML?

I have a few entries in a brand new dictionary created only with the new versions. I tried to enter from a .txt file using the format /w::/c::/d (among other formats). I saw no new entries into the dictionary however.
An entry looked like:
word::Noun::definition
At first I thought it might be because I was using Greek characters, so I switched to Latin alphabet, and I tried both UTF-8 and ANSI encodings if that makes a difference.

On some positive notes, I noticed that sound classes are applied automatically if a phoneme contains the features that define that class.
Also, my use of Greek characters is well supported for spelling side.

[faiuwle]

I have a few entries in a brand new dictionary created only with the new versions. I tried to enter from a .txt file using the format /w::/c::/d (among other formats). I saw no new entries into the dictionary however.
An entry looked like:
word::Noun::definition
At first I thought it might be because I was using Greek characters, so I switched to Latin alphabet, and I tried both UTF-8 and ANSI encodings if that makes a difference.

You're right, I broke it. It'll be fixed in the next update. Speaking of character encoding, though, you'll need to set Settings -> Force Unicode to import UTF-8 files correctly if you're on Windows (which seems to always assume they aren't UTF-8).

On some positive notes, I noticed that sound classes are applied automatically if a phoneme contains the features that define that class.

There's actually no difference in the database - a phoneme/word is a member of a class iff it contains all the requisite features.