Announcements

Watch this space for important information on planned twitch streams, updates and more

How to : Validate a good content mission from the mission builder

17 posts / 0 new
Last post
TitansCity
TitansCity's picture
Offline
Last seen: 1 day 6 hours ago
11th Anniversary Badge
Joined: 10/28/2013 - 02:09
How to : Validate a good content mission from the mission builder

I thought about the mission builder and i remind how the missions list in CoX was either awfull and didn't reflect really the quality of the mission.
I don't know the way the mission was validate before being push in frontline to players so as they can play it. So, this is only a reflexion of a system. Maybe it will be too hard to build it but i wanted to share this idea because i think the validation must not be done like something not important if CoT "must" have good contents so as players dislike using the mission creator.

But, to avoid things like that, to avoid having some "not finished" missions, "impossible to end" missions, or just bad quality missions, i think about How validate the missions. Far from a form to fill or voting throught some stars or hearts, i wonder : And what about if the validation is done throug a test by the players who wants to be validators ?

  • Ok, i open the mission builder and decide; not to build a mission but to validate a mission. I just click on "Try a mission to validate" (or something else of course ^^)
  • Context : the mission is build, the form of the mission is filled at least to 90% (general informations like name, name of the author, note of the author, title, ennemies you will encounter, environnement type, goal, mission type, class targeted ?, etc etc)
  • The mission i will choose is stored in a backlog of missions to approve.
  • I pick one which form is filled and help me to make a choose. Ok, i want to try something in which i could face somme Regency foe and/or Scorpion ennemies.
  • Nice ! "The Regency stole an artifact in a museum (the Nathaniel Clark's House in Clarkstown) and will sell it to the Scorpions". This mission is waiting for an approval and i can see that 12 players on 20 had already test it. I have no more information on the test (yes, this is a blind test ^^).
  • I launch the mission and end it (or, if i can't end it, i can click on the "exit mission" button)
  • Now, i have a form to fill. Not a big one, but one which help to validate and classified the mission. There are 2 parts :

    Global validation of the mission :

    • Does the mission answer to the general information ?
    • Do the main ennemies are really Regency and Scoprions ?
    • Does the environnement type is correct ?
    • Is the goal clear or was it not so clear to understand during the mission ?
    • To your opinion, which archetype can handle with this mission : all ? only some -> check the corresponding boxes. Etc, etc.

    My opinion on the mission :

    • What is funny ?
    • do i enjoy the story ?
    • Texts are good and without errors ?
    • Does the text was to violent ?
    • I leave a comment on my feeling on that mission so as the author can improve it and modify it.
    • Was it too easy or too hard ? (graduation from easy, not so really easy, pretty hard, too hard, hardcore and i loved it) <-- the archetype and style could be use to determine the final result of this point.

  • I submit my form
  • Now, i can see that the 12 players gave a average rate of B on a rate from A to E (or 16/20 if you prefere with a rate from 0 to 20). I can read the comments of other players
  • When i return in the backlog of mission, this one is not available anymore since it is still in approval until the 20 players test it. But, another player could see it to test it.
  • [*]When the 20 players have test it, the mission is available in the mission list of the playable missions.

    How to determine if a mission id available to the list ?
    Some conditions and average rate to reach could determine if the mission can be available.
    For example : on 20 players, 18 found the mission was to violent. A too violent mission must not be available because of the T for Teen. So the mission is rejected, the author can modify his mission and/or text. Once it is modified, the mission returns in the backlog of mission to approve but, this time, it just need the validation of 10 players and the form is only about the text or mission violence.

    Foradain
    Foradain's picture
    Offline
    Last seen: 15 hours 44 min ago
    kickstarter11th Anniversary Badge
    Joined: 10/25/2013 - 21:06
    Crowdsourced editing ^_^.

    Crowdsourced editing ^_^. And of course, the devs (or persons designated by the devs) can do spot checks to see if anything is getting through that shouldn't, and perhaps tweak the survey phrasing, or increase the number of testers required.

    Foradain, Mage of Phoenix Rising.
    Foradain's Character Conclave
    Avatar courtesy of Satellite Nine.
    If you can't see an image I've posted, please let me know!

    TitansCity
    TitansCity's picture
    Offline
    Last seen: 1 day 6 hours ago
    11th Anniversary Badge
    Joined: 10/28/2013 - 02:09
    Yes, but i think this could

    Yes, but i think this could really be automatized :)

    Huckleberry
    Huckleberry's picture
    Offline
    Last seen: 1 hour 1 min ago
    Joined: 01/03/2016 - 08:39
    I like the idea that a

    I like the idea that a mission is not available for public consumption until X amount of validation has been performed.

    But this creates a scenario of players not wanting to validate content if there is content already validated available to them. Until there isn't. Then you will basically be forcing players to validate content if they want new missions. Most players aren't going to want to have to do this. But then, you really don't need most do you? You really only need the select voluntary few who are willing to run the content because they want to help make it better. Will that be enough?

    I think it would be a good idea to reward players who perform validation reviews. Heck, the game could even make it an achievement with a badge reward. Sure, you'll get all the undesireable input that happens when you make it "mandatory" but at least you will get the input. Rather that languishing at 17/20 validators forever because people would rather run fully tested content than unfinished or untested content.


    I like to take your ideas and supersize them. This isn't criticism, it is flattery. I come with nothing but good will and a spirit of team-building. If you take what I write any other way, that is probably just because I wasn't very clear.
    Project_Hero
    Project_Hero's picture
    Online
    Last seen: 10 min 16 sec ago
    Joined: 10/09/2014 - 11:21
    If you give rewards for it

    If you give rewards for it then people might just skip through the validation part (clicking highest or lowest clickies, writing gibberish in any long form parts) just to get the rewards.

    Things have shown that when you leave things completely up to the community to police it tends to not end well (coughSteamGreenlightcough)

    That said I do not have a better solution at this time.

    "Let the past die. Kill it if you have to."

    TitansCity
    TitansCity's picture
    Offline
    Last seen: 1 day 6 hours ago
    11th Anniversary Badge
    Joined: 10/28/2013 - 02:09
    There are some solutions for

    There are some solutions for that :)

    1. The reward is available and taken in account only when more than 200 missions are completely validated ? (which means if the mission is too hard, the reward is not available. But if the mission is too hard, 20 people will exit the mission and tellthe mission is too difficult and the mission will be not validated... Maybe, the reward could be available or taken in account only when all the form is filled ? There is, here, a condition to make the reward not so easy to have so as to keep people on validating missions)
    I think it could be anought to have people who wants to validate missions and for many reasons. The creators will probably take parts in validating missions, people who wants to have another source of playing time as well. At the beginning, this system could take some time to validate the missions and over time, the content will grow up unless the missions are rated to be deactivated after a certain period of time regarding the score given by the players (E.G : less than 2 hearts will be deactivated within 3 months, less than 3 hearts within 6 months, less than 4 hearts within 1 year, 4 & 5 hearts stay activated until the rate is lower). i'm thinking in the same time i'm writing :D Sorry for that.

    2. People who wants to have good contents will validate those missions, for sure and seriously but people who wants to have mission in advance or wants to try something new or want to have some new ideas could use this too. Also, it could be something to "advertise" in a corner of the UI of the mission creator/mission launcher with an "Why not try the pending missions ? There is X mission to validate, try them and give your opinion !".

    Project_Hero
    Project_Hero's picture
    Online
    Last seen: 10 min 16 sec ago
    Joined: 10/09/2014 - 11:21
    If there's a reward to be had

    If there's a reward to be had those who want the reward will validate missions. Not just those who want good missions. It'd probably end up with a group flooding the mission thing with easy completion missions so that they can all do them, validate them, and then get the rewards. Even if you need to fill out a form for it, it'll only take a few clicks and mashing buttons on the keyboard to do (improperly).

    It'd be better to have the validation process garner zero rewards so that only those interested in validating missions will do it. But that still won't stop people who make farming missions from getting a group together to validate their farming missions. Or those who make missions in poor taste or quality (ie incomplete messes) from doing the same.

    "Let the past die. Kill it if you have to."

    Huckleberry
    Huckleberry's picture
    Offline
    Last seen: 1 hour 1 min ago
    Joined: 01/03/2016 - 08:39
    Project_Hero wrote:
    Project_Hero wrote:

    If there's a reward to be had those who want the reward will validate missions. Not just those who want good missions. It'd probably end up with a group flooding the mission thing with easy completion missions so that they can all do them, validate them, and then get the rewards. Even if you need to fill out a form for it, it'll only take a few clicks and mashing buttons on the keyboard to do (improperly).
    It'd be better to have the validation process garner zero rewards so that only those interested in validating missions will do it. But that still won't stop people who make farming missions from getting a group together to validate their farming missions. Or those who make missions in poor taste or quality (ie incomplete messes) from doing the same.

    This is a really excellent point. A reward system will create a market for easily validated missions, flooding the system with trash and keeping the excellent, in-depth missions from getting validated.


    I like to take your ideas and supersize them. This isn't criticism, it is flattery. I come with nothing but good will and a spirit of team-building. If you take what I write any other way, that is probably just because I wasn't very clear.
    Fireheart
    Fireheart's picture
    Offline
    Last seen: 1 day 20 hours ago
    11th Anniversary Badge
    Joined: 10/05/2013 - 13:45
    There were players in CoH and

    There were players in CoH and I expect in CoT, who actually Enjoy Playing/Validating/Critiquing user-made-content. In the Beta, this sort of thing should be run on Everything. I do think it should/could be mostly automated, in that Everyone who runs through a mission in Beta should be asked to critique it afterward.

    Be Well!
    Fireheart

    Huckleberry
    Huckleberry's picture
    Offline
    Last seen: 1 hour 1 min ago
    Joined: 01/03/2016 - 08:39
    I actually had an idea to

    I actually had an idea to encourage players to rate content and it involved a reputation system where the reviewers were themselves reviewed by both other reviewers (as in "this was a helpful review") and rewarded by the content creators themselves.

    But that idea was for content that was already ready for public consumption, not for pre-public validation efforts.

    I think there is a way to capitalize on that idea for our discussion of validating content. If we take the idea of the content creators grading the reviews we could make this work. I would make it anonymous so that the content creator does not know who wrote what reviews. This will keep some of the nepotism down. Then the reviewers who get graded as providing the most helpful reviews will get some points towards a reward/achievement.
    It won't stop the people who deliberately game the system using other means of communication to farm for the rewards, but then again will anything really keep people from taking advantage of anything if they put their minds to it? It also won't stop some content creators from tossing rewards only to the people who write glowing reviews. But remember these are the pre-public validation reviews, so it really doesn't do the creator any good to only reward non-critical reviews if they end up getting lambasted once the content goes public.

    The checks and balances are there, I think. But because it is my idea, I'm a bit too close to it for proper perspective. Can anyone else see flaws in reviewers' reviews being anonymously graded by the content creators during the validation stage?


    I like to take your ideas and supersize them. This isn't criticism, it is flattery. I come with nothing but good will and a spirit of team-building. If you take what I write any other way, that is probably just because I wasn't very clear.
    Project_Hero
    Project_Hero's picture
    Online
    Last seen: 10 min 16 sec ago
    Joined: 10/09/2014 - 11:21
    Maybe instead of a way to

    Maybe instead of a way to validate newly made missions focus should more be on removing troublesome missions? Though that might still have to be community driven...

    This is a tough issue.

    "Let the past die. Kill it if you have to."

    TitansCity
    TitansCity's picture
    Offline
    Last seen: 1 day 6 hours ago
    11th Anniversary Badge
    Joined: 10/28/2013 - 02:09
    After mal That ideas, it

    After mal That ideas, it comes To me that reviewing/rating/etc should not take too much actions for the user.
    If we thought about this anonymous system (which an idea to really keep in mind ), this ask the creator To come back to validate or rate the feedback. I like this idea but i dislike the fact To come back to rate all the reviews.

    Cobalt Azurean
    Cobalt Azurean's picture
    Offline
    Last seen: 2 hours 43 min ago
    kickstarter11th Anniversary Badge
    Joined: 10/03/2013 - 16:39
    Huckleberry wrote:
    Huckleberry wrote:

    Project_Hero wrote:
    If there's a reward to be had those who want the reward will validate missions. Not just those who want good missions. It'd probably end up with a group flooding the mission thing with easy completion missions so that they can all do them, validate them, and then get the rewards. Even if you need to fill out a form for it, it'll only take a few clicks and mashing buttons on the keyboard to do (improperly).
    It'd be better to have the validation process garner zero rewards so that only those interested in validating missions will do it. But that still won't stop people who make farming missions from getting a group together to validate their farming missions. Or those who make missions in poor taste or quality (ie incomplete messes) from doing the same.
    This is a really excellent point. A reward system will create a market for easily validated missions, flooding the system with trash and keeping the excellent, in-depth missions from getting validated.

    Pretty similar to what happened to Mission Architect. Too many of the "most played" and "highest rated" were just efficient ticket farms.

    Huckleberry
    Huckleberry's picture
    Offline
    Last seen: 1 hour 1 min ago
    Joined: 01/03/2016 - 08:39
    TitansCity wrote:
    TitansCity wrote:

    After mal That ideas, it comes To me that reviewing/rating/etc should not take too much actions for the user.
    If we thought about this anonymous system (which an idea to really keep in mind ), this ask the creator To come back to validate or rate the feedback. I like this idea but i dislike the fact To come back to rate all the reviews.

    I think that we are not asking the creator to do anything the creator would not already be doing.

    If the creation is going to be validated, I assume that implies some sort of feedback mechanism to the creator; and if the creator wants feedback on his or her creation, then it only makes sense that the creator would look at this feedback. And when the creator reads the feedback all he or she should have to do is click one of three buttons: "not helpful", "helpful" or "very helpful" I think that's all that is needed here.


    I like to take your ideas and supersize them. This isn't criticism, it is flattery. I come with nothing but good will and a spirit of team-building. If you take what I write any other way, that is probably just because I wasn't very clear.
    Fireheart
    Fireheart's picture
    Offline
    Last seen: 1 day 20 hours ago
    11th Anniversary Badge
    Joined: 10/05/2013 - 13:45
    In the Beta and probably

    In the Beta and probably afterward too, it could all just be database-driven. I do think, on the Developer side, each report should be tagged with a username, just to highlight who is giving useful and useless reviews. THAT would be where the Reputation could come in. Content-makers could rate the reviews themselves, but not see who posted them. However, the Devs could see if there's a recursive pattern of friends high-rating each other's content to generate false-positives.

    Be Well!
    Fireheart

    TitansCity
    TitansCity's picture
    Offline
    Last seen: 1 day 6 hours ago
    11th Anniversary Badge
    Joined: 10/28/2013 - 02:09
    Fireheart wrote:
    Fireheart wrote:

    In the Beta and probably afterward too, it could all just be database-driven. I do think, on the Developer side, each report should be tagged with a username, just to highlight who is giving useful and useless reviews. THAT would be where the Reputation could come in. Content-makers could rate the reviews themselves, but not see who posted them. However, the Devs could see if there's a recursive pattern of friends high-rating each other's content to generate false-positives.

    I wonder... rating the helpfulness of feedbacks is really personnal, as rating the mission (maybe i'll dislike the regency faction and there is too much regency foes on my opinion in that mission, so i'll give a bad feedback). Maybe it could be more efficient to rate some objectives criterions ? Like : corresponding faction as announced, the story fits with what was announced… and then, maybe 2 or 3 personnal feedback like did you enjoy this mission ? Do you think it must be deployed to public ? And, i think, the mission must be anonymous, so as the mission will be not rated compared to the author or something like that... and the feddback as well :)

    I like the reputation system for creating and rating the creations. Moreover, skipping the mission through an "exit" button could open a mandatory pop up to tell why the exit button was used ("i can't finish", "there are too much ennemies for me","i can't end the mission", "i don't want to finish it" etc) and that will not be taken in account for the reputation. Only the complete filled form could be taken in account for this reputation ?

    Planet10
    Planet10's picture
    Offline
    Last seen: 4 days 7 hours ago
    Joined: 03/23/2016 - 17:21
    Crowdsourcing the review

    Crowdsourcing the review process is part of what should happen. Review by the MWM team should still happen no matter what.

    Another facet is to require a person who wants to create a mission to have reviewed someone else's mission prior to being able to submit their own. There would have to be some bootstrapping to get the process rolling, but it will give the creative types exposure to other's ideas. Hopefully that will grow a community that will provide feedback to make better content.