-81

We are happy to announce this year's Developer Survey is officially open - take the survey here.

There are lots of new questions, classic questions, some returning questions and many updates based on the great feedback we received here. We encourage you to take the survey today, and share it with your friends and coworkers who could be directly or indirectly working with technologies related to programming, cloud capabilities, knowledge capture or anything new/emerging.

We will be sharing all of the results in a few months like we usually do. Help us make this the best survey yet by filling it out before the survey closes!

31
  • 70
    Just some feedback--most of the questions seem to be AI-focused rather than developer-focused. I haven't seen any questions about the tools or languages I use or am interested in using in the future. I have seen a bunch of questions about AI, even after saying in 3 or 4 different responses that "I don't use AI". Many of the questions seem to be about "using AI agents" because they ask very odd questions about "context" which don't make sense if you aren't using AI tools. Commented Jun 23 at 15:23
  • 58
    I don't have a grade in mind for the survey based on this but it seems y'all have forgotten/missed the point of surveying SO users: we're programmers and we want to talk about programming tools and programming languages. What lang/tool we use, what we want to use in the future, what we dread using, etc. That's what's fun, and what drives engagement for the survey. Not a laundry list of whatever AI tools no one has heard of but that companies have paid you to get some airtime in front of SO users (just my guess as to why there were literally 3 or 4 pages of no-name AI tool questions). Commented Jun 23 at 15:24
  • 45
    Slight correction--after 1-2 dozen pages of AI questions, I finally got to the end of the survey where you asked me what tools and languages I like to use; the survey would've been better without those 1-2 dozen pages of AI questions. Commented Jun 23 at 15:42
  • 22
    I've been doing this survey for years, I don't think I've ever skipped questions before til today WAAAY too much on AI. I won't bother answering next year if the survey is like this again. Commented Jun 23 at 16:23
  • 11
    I respond every year, but this time I just couldn’t respond to all these very specific questions about AI use so I skimmed over most of the second half of the survey. I don’t find that very respectful of your users given that we have no compensation for these reponses. Commented Jun 23 at 17:14
  • 48
    After the massive backlash against all the AI questions in last year's survey, what made you decide to double down on it this year? Commented Jun 23 at 17:43
  • 14
    I skipped probably half of this year's survey for the first time, none of the AI questions apply to me. Give us an opt-out box at the beginning so we can fast forward through all the slop questions. And then when I get to the question about how I learn the answer to a problem, "reading the documentation" really needs to be an option. Commented Jun 23 at 18:01
  • 10
    This survey is incredibly sloppy. Massive amounts of wasted whitespace, which made single questions overflow the screen height; Questions with missing options; Inconsistent formatting; And worst of all, massive focus on AI... Commented Jun 23 at 20:55
  • 9
    Stack Overflow are still really good at listening to feedback from the community, I see. Commented Jun 23 at 23:19
  • 11
    The survey has officially jumped the shark. I may not even bother reviewing the results this year let alone take the survey again next year. The developer survey used to be a huge value to the community. Now it's just a way for SE to get free product research. It's a real shame. Commented Jun 24 at 0:41
  • 25
    Was any of the feedback from meta.stackoverflow.com/questions/438884 actually taken into account? At all, whatsoever? Commented Jun 24 at 0:50
  • 18
    Did anyone on staff attempt to take the survey before it went live? There are many responses already pointing out flaws that I would consider obvious and embarrassing if I were in the position of administering a survey. Commented Jun 24 at 1:00
  • 25
    Good God, what happened to the Developer Survey? It literally read like it was the output of a prompt that said "Prepare a 2026 Stackoverflow Developer Survey in Qualtrics format" while the model was in full hallucination mode? Questions were unanswerable, presumed there must be an AI reason for every decision to the point I felt the survey no longer understood the concepts of using a tool "because it meets project requirements" or deciding to learn information for the reasons of "need or necessity". Commented Jun 24 at 6:52
  • 20
    Based on the answers here, and the comments, I would suggest this is going to be the first survey I'm not only going to not answer in many years, but not even bother opening. If it's going to, again, ram AI questions down my throat, as someone that doesn't use AI, it's completely irrelevant to me. The last one kept on asking my about my AI usage, when many I couldn't even answer because they weren't relevant. It would be like asking someone "Do you drive to work?" and when they respond "I don't own a car." you follow with "And what type of car do you own? Petrol, Diesel or Electric?" Commented Jun 24 at 10:39
  • 9
    One other thing: there was somewhere a question on how to judge A.I. generated answers. I only accept A.I. generated answers if I can verify them. (That was not one of the possibilities) Commented Jun 24 at 11:03

29 Answers 29

Reset to default
95

The survey is far too long. I spent over half an hour on it before bailing towards the end. There's no indicator of how many questions are in the survey, but I must have answered 50 questions, some of which consisted of up to 95 checkboxes to tick.

The survey's primary aim appears to be collecting as much data as possible to justify the Stack Overflow Internal product to customers and shareholders rather than improve what's left of Q&A. Many questions are redundant, circling around the same handful of topics.

I'm not familiar with SO Internal directly, but I saw Jody Bailey's recent talk at O11ycon which could be summarized as "internal organization context is disorganized and unreliable and devs don't trust AI, but our SO Internal product solves this by sourcing authoritative, timely, trusted information".

Dozens of questions in this survey revolved around AI trust, attribution, timeliness, and other concerns that are basically irrelevant to Q&A (since GPT content is banned) but were key value propositions in the SO Internal talk.

Questions like "When working on a project or task, which types of context are most important for you to have?", "Which of the following make it difficult to get the context you need for work?", "How often do you discover important work context only after you have already started or completed a task?", "Which of the following do you do to manage or prepare context for your work?" are unrelated to SO Q&A and straight out of the SOI talk's slides. (These are just a few I happened to pick; there are dozens of questions like this in the survey).

Setting Q&A aside, questions hyper-focused on minutiae don't provide much insight to the broader state of software engineering, either. I don't recall prior years' surveys drilling heavily into details like this.

It feels like Q&A is a sideline (or nonexistent) concern for the company at this point. I'd prefer a survey that acknowledges the current state of public Q&A and seeks feedback on proposals of how to carry it forward, or raises interesting questions about software engineering today, rather than a way to collect data for SO's enterprise products.

6 or 7 carefully-chosen questions would be sufficient to capture an understanding of which AI tools developers use and how they use them. For developers who don't use AI, the survey could auto-skip the remaining AI questions.

As it stands, I don't think you can trust the data you get from this survey. It's biased towards the narrow subset of people who bother to complete it, and among completions, I bet a non-trivial amount of random checkbox rage-ticking occurred.

17
  • 47
    Effectively, what was once the premier survey for understanding developers has turned into... something else. Commented Jun 23 at 16:50
  • 2
    I disagree with most of this comment, but yes, many of the questions are redundant, or could be entirely derived from previous questions. Commented Jun 23 at 17:13
  • 2
    @davidalayachew what do you disagree with and why? Commented Jun 23 at 17:16
  • 3
    @ggorlen We'd need a chat room to answer properly, but in short, it feels like you are ascribing intention without the context necessary to do so. Moreover, I feel like there are way more likely reasons for the ones you gave. I'm not really pressed to expand on them here, so I can delete my comment if you prefer. Commented Jun 23 at 17:21
  • @davidalayachew no chat necessary, that clarifies matters--thanks. The company's intent is clear from their actions. What else would motivate this particular set of Q&A questions? Clearly not the Q&A. The only other product they have 100% lines up with the data points they're collecting here, unless I'm missing something. Commented Jun 23 at 17:24
  • 2
    @ggorlen Call it a personal pet peeve, but I am against making assumptions, regardless of how likely they may be. There is a phrase about assuming the best until proven to be the worst, but I don't remember it at the moment. And of course, I am not at all trying to say that this company treats its people well. But, since there are many new faces and new folks, I am trying to give them a shot and see if they do right, even for the folks that messed up badly before and are still here. I wasn't involved, but I do remember reading about the Monica debacle. So, the ill-will is understandable. Commented Jun 23 at 17:38
  • 1
    @davidalayachew I suggest you carefully re-read my answer. I don't think your usage of "ill will", "worst" and "best" is warranted here. I'm not commenting on how well SO treats its employees or ascribing nefarious intent to anyone, per se. I'm pointing out that the evidence overwhelmingly indicates SO has aligned this survey and overall biz priorities with their Internal offering, not Q&A. Maybe that's a good thing for some people, and maybe there's nothing "wrong" with them doing this if they feel it genuinely provides value. I happen to disagree with their direction. Commented Jun 23 at 17:47
  • 1
    @davidalayachew The current situation is this: Q&A is dead, SO tried to launch a beta experiment to replace it (basically a reddit rewrite that removed all curation), which failed. Concurrent to Q&A, SO has an internal enterprise offering, and now that's all that's left to bring in money, since Q&A engagement has evaporated. SO can also presumably sell Q&A data and has removed the data dump. How to salvage human Q&A? Is there even any value to Q&A in the agentic era? I don't have an answer, and neither does SO so far, for better or worse. I'd like to see a survey that attempts to address this. Commented Jun 23 at 18:01
  • 1
    @davidalayachew The Meta Room already exists for general chat or general conversation regarding Meta posts Commented Jun 23 at 18:07
  • 6
    "I'm not familiar with SO Internal, but I saw Jody Bailey's recent talk at O11ycon" — are ordinary programmers supposed to have any idea what that event name even means? Commented Jun 24 at 0:52
  • 33
    The answer of "[x] I do not use AI in my work." seems to be completely ignored because it preceded all of the "Context" questions. It was the most nonsensical SO Developer's Survey I've taken, and I've taken more than a decade worth. It was overly repetitive which contributed to it being way too long. It really just left me disappointed if this is what tech has become. To the point I even put in the survey comments I was going to raise the issue on meta.stackoverflow -- but you all beat me to it :) Commented Jun 24 at 7:03
  • 7
    @DavidC.Rankin wait, so "context" there was used as in "LLM tokens context", not "scope/surrounding circumstances"? Then my answers to context questions are completely off, I have selected some items, but I don't feed any work "context" to any LLM, so do not encounter any such issues... Commented Jun 24 at 16:35
  • 2
    @DavidC.Rankin No, I mean, I read and answered those questions with general meaning of "context", and was surprised when your comment was acknowledged as a bug. Like, I can answer "don't use AI", but still have troubles with gathering work context for myself (I do in fact). So it should've been pointed out somewhere if they wanted "context" to mean "llm context" - it's really not obvious for me... Commented Jun 24 at 19:46
  • 8
    @DavidC.Rankin As someone who doesn't use AI, I didn't even realize that "context" was an AI buzzword, so I answered those questions assuming the normal meaning of the term. If a lot of other people made the same mistake, that's certainly going to affect what little integrity the results may have had. Commented Jun 24 at 22:35
  • 1
    @STerliakov sorry, my misunderstanding. Yes, that blew me away. The "context" there, best of my understanding referred to the number and type of tokens used by whatever model you are using, e.g. permanent, temporary, etc. and that these bot/agent things have a budget of them you can run out of. That's just from an article I'd read and the limit of my understanding of that stuff. So I was really surprised for the Survey to keep repeatedly dunning me with questions about context after I told it I didn't use AI at work. It just seemed pointless, and presumes everybody is an AI nut. Commented Jun 25 at 5:17
72

How is anyone expected to answer this?

enter image description here

An honest count would land me somewhere between 20 and 40, depending on what counts as a "tool".

8
  • 14
    I came here to post the same thing. I couldn't even tell what they were trying to ask. It's such a vague question with such specific exclusions. Commented Jun 24 at 0:43
  • 3
    They ask something similar every year. I've never known if "software tool" means any tool that's implemented in software, or a tool used for developing software. Do you want me to count my VOIP software, or just my development-related stuff? Does emacs count as a tool or an operating system? As with most of these types of questions, answering would be much easier if they explained how the answers would be interpreted. Commented Jun 24 at 1:20
  • 3
    My FAVORITE question! Given the Count... and then Do not count... instructions, the only thing that popped into my mind after reading the question was WTF?? Okay, and for completeness, I also thought "Beam me up Scotty, there is no intelligent life left here..." I've taken at least 10 or so of the surveys, but I will not be wasting my time on the 2027 survey. Commented Jun 24 at 7:10
  • 4
    i just answered 42 Commented Jun 24 at 14:40
  • 3
    Tools count but programming languages don't. Does sed count? What about vim? Commented Jun 24 at 18:44
  • 3
    Does Stack Overflow count? How about Super User or Database Administrators? If I used 2 or 3 of them, is that 1 or 2/3, as they're all part of the Stack Network. /Shrug. Commented Jun 24 at 21:54
  • I didn't count awk or sed due to the instruction not to count "multiple interchangeable utilities..."'. It was just a bad question all around. Just read the Count... part slowly and try and make sense of it. Then the Do not count... part the same way. At least of my read there wasn't much left over to actually count, so I just skipped it. Commented Jun 25 at 5:22
  • I'd say about 30 Commented Jun 27 at 8:30
61

These meta threads are so meaningful and listening to community feedback is oh-so important and that's totally not a blatant lie.

  • SO: what would you like to change about the survey this year?
  • Community-to-a-man: don't make it all about AI.
  • SO: hey I know, lets make the survey all about AI!
  • SO: how did you like this year's survey?
  • Community-to-a-man: I hated it, you made the survey all about AI!
  • SO: gasp! they didn't like it, how could this be!
3
  • 9
    Yeah, but that's the state of the entirety SO at this point. Commented Jun 24 at 18:47
  • 7
    I'd change the last point to "everyone loved it being about AI" ... Because SO seems to just live in their dream world and are completely oblivious to community feedback. Commented Jun 25 at 12:04
  • If you count the questions in the survey repo. Of the 110 questions they ask almost 50 are in the AI category alone. Commented Jun 25 at 20:34
40

This question only allows one option: enter image description here

But the next question seems to imply it should've been a multiselect: enter image description here

Also, what's with the insane amount of wasted space on that first question?

40

This question needs an "I can verify the correctness of the information." answer.

enter image description here

6
  • 12
    Also, I can't understand what "The information has many citations, sources themselves don't matter as much as the ability to track down the origin" is supposed to mean. (also, far less relevantly, it's a comma splice) Commented Jun 24 at 12:14
  • 12
    "...has many citations..." would be a best choice for me, but they chose to continue with "sources themselves don't matter as much" - what? Like, am I supposed to just count citations and say "wow, they added five links, that's credible"? The whole point of citation is my ability to read the source, usually ignoring whatever AI extracted from it... Commented Jun 24 at 16:29
  • 1
    Exactly. I ended up checking "has many citations", but the sources matter very much to me. Commented Jun 24 at 18:45
  • 4
    There are crucial difference between "I can track down the origin" and/or "The information has many citations" compared to "I can verify the correctness of the information". 1. Tracking down the origin doesn't mean something is correct. E.g., somebody citing W3Schools could have faithfully re-transmitted wrong information. 2. For some topics I can verify the correctness without needing to be given sources. If I'm an expert, I might already know if it's correct, or I might know where to look it up. Commented Jun 25 at 8:29
  • "When I implement and test it myself." Really should have been an option. Commented Jun 26 at 14:06
  • 1
    The author of this question does not understand the SO target market as I see it. Commented Jun 27 at 16:45
37

Some feedback: I feel like this question would benefit from clarifying which unit of time to use.

In a typical week, how much time do you spend looking for information you need to do your work?

6
  • 6
    Yeah, I ended up leaving that one blank because it didn't like "1 to 8 hours" which was my first attempt. Commented Jun 23 at 15:33
  • 1
    Can one of the staff members comment here with the intended unit? Then, while the survey is being updated, we can continue. At the moment, the survey is just sitting open in another tab. Commented Jun 23 at 16:01
  • 18
    This now seems to have been fixed: "Please provide a whole number representing hours per week, rounding up to the nearest whole number if needed." Commented Jun 23 at 16:14
  • @MiloP Not yet for me, but I will continue and just assume that they wanted hours. Commented Jun 23 at 16:27
  • Not fixed for me, I still see the the text as it is on the screenshot. Commented Jun 23 at 17:50
  • 3
    This question is actually kind of interesting to me, because ever since the AI craze has made finding useful information a whole lot harder, the amount of time I spend needing to search has decreased drastically. I commit more to memory now or capture it in some kind of test program or document, because I have to. Information is rotting away every day. Commented Jun 24 at 10:00
35

Important note on ad blockers and cloud security tools: ad blockers and cloud security tools can sometimes block essential scripts or elements needed for the survey to function correctly. To avoid any potential error messages, delays in the survey appearing or disruptions, please consider temporarily pausing your cloud security or ad-blocking plugin, or specifically unblocking Qualtrics while you take the survey.

Why? Or more accurately, why does my ad blocker prevent part of your survey from loading?

2
  • 9
    Most likely, because the site is hosted at stackoverflow.co but the survey is powered by Qualtrics and also uses recapcha. Cross-domain stuff can get blocked by strict security software. Commented Jun 24 at 1:52
  • 2
    I don't recall seeing this, and I use uBlock Origin on Firefox. Or maybe it was shown and I just didn't notice it. It didn't seem to impact my ability to use the survey. Maybe what they meant was tracking cookie blockers instead of ad blockers? E.g. Ghostery and its ilk Commented Jun 25 at 19:44
33

These surveys seem to be written assuming that everyone on Stack Overflow has a job in software development, but for many users (like me) that isn't true. For example, many people code as a hobby outside of their job; many people are starting their career and making a portfolio; and many people are learning to code in school.

If you want the survey to reflect everybody using Stack Overflow then I'm not sure why these groups of people have been forgotten. Here are some examples of where I didn't feel the questions were intended for people in my position:

(Side note: I've double-checked my answers so I know this isn't a case of me answering questions wrong and then getting interesting results.)

  • I code personal projects every day, but not part of a job - however this question doesn't seem to like that:

    Question: Are you someone who creates code? Responses: Create as part of job; Create occasionally; Review but don't create code; Manage people who create code; Don't create code

  • And this question won't let me answer "0", even though that's the correct answer:

    The question asks how many years I've been working professionally in total, but it doesn't allow the answer zero

I'm also concerned about the readability of the language used in the survey. This question stands out in particular:

In the last 12 months, has your role at work, school or in personal projects intersected with any of the following types of work involving technology? Consider what you used in each context. If you had more than one role in a context, refer to all roles within that setting that you held in the last 12 months. Select all that apply.

3
  • 3
    Yeah, that (last) question definitely came from a McKinsey consultant who specializes in fluffing HR department heads. Commented Jun 25 at 18:18
  • Debatably, zero isn't a whole number so I can kind of see that based on what they said, but yeah they should allow zero. Commented Jun 25 at 19:43
  • “And it’s whole with a W, for those of you that have a problem.” Commented Jun 26 at 9:45
31

Someone recommends this every year, but please start numbering the questions. It will make it significantly easier to discuss them unambiguously.

On many questions, the "None of these" option is incorrectly rendered as a checkbox and not a mutually-exclusive radio button. That lets you select "None" while also selecting other options, resulting in a contradictory answer.

In a number of places, you refer to a tool named "Make". Is this referring to make.com the online automation platform, or GNU Make, the classic command-line tool that has powered toolchains for the last half century? Either would be valid answers for the questions where this is used.

The "Do you try to avoid using AI tools" question is lacking what (IMO) would be common reasons that don't fit with the current options:

  • Restricted by company/school policy
  • Legal concerns
  • Regulatory requirements
  • Cost
  • Quality concerns
  • They don't actually help me

The "Which actions do you turn to first when trying to find answers to a problem?" question is surprisingly missing the obvious response: read the official documentation.

On a side note, I applaud you for posting the survey itself publicly on Github for review, comment, and contribution. I hope you continue to do this in the future. Next time, though, please consider posting the questions more than five days before the survey goes live, plus advertising it in the same places across the network where you advertise the survey itself. I feel like most (if not all) of the issues pointed out in all of the answers posted so far could have been caught and fixed ahead of time if you had three or four pairs of disinterested eyeballs proofreading it. As it stands, the only contributions to your repo were three discussions started after the survey went live, because nobody knew it existed or was given an opportunity to contribute before it was published.

2
  • 1
    Numbering the questions may be non-trivial, given that some questions are asked conditionally based on other responses. Commented Jun 26 at 22:05
  • 2
    @RyanM - The numbers don't have to be sequential or continuous. Jumping from question 2 to 18 to 24 is perfectly fine. Just so each question has a unique ID of some sort. Commented yesterday
23

I understand that it's too late to change now, but in the "select at most N" questions, the max number of answers felt too limiting, and I couldn't indicate the relative importance of each answer.

I would prefer all those to use the "rate each choice" format that was used for some other questions (the matrix of radio buttons).

4
  • 4
    Yep, I selected like 15 options in one of these "select top 3 or 5" before realizing I was limited. Commented Jun 23 at 18:14
  • 11
    It certainly doesn't help that many of these don't really distinct answers but a ton of equally important choices. "What are the most important parts of your car? Front wheels, back wheels, or steering wheel – choose at most 2." Commented Jun 23 at 20:03
  • Why am I allowed to answer more questions than the max? What happens if I do? Commented Jun 24 at 18:48
  • 2
    @LyndonGingerich It errors when you try to click "next". Commented Jun 24 at 19:43
23

Hiding the "Which development environments and code editors did you use regularly over the past year?" question behind "I used a development environment (IDE) to write code or interact with agent-assisted developer tools" is really scummy. I don't remember that from previous years.

Most people wouldn't consider Neovim/Nano/Notepad++/etc an IDE so they might unknowingly click right by it.

The current phrasing of this year's survey is a surefire way to make sure editors are underrepresented in the final results. Shame on you, Stack Overflow.

Sources:

23

Several of the questions would have benefited from having an option "I'm not using/do not want to use any of these AI tools."

How can you tell whether someone is just skipping the question versus answering the question by selecting none of the options?

New contributor
BlueQuark is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
1
  • 12
    Several of the questions would have benefitted from not being shown at all to users who have already selected "I do not use AI tools" in an earlier question. Commented Jun 25 at 19:39
22

When prompted to accept ToS and Privacy Policy, when selecting I do not agree (selecting this will end the survey) nothing happens - after clicking Next page > the survey simply starts.

Tested on Firefox and Edge, without the adblockers.

3
  • 8
    My word, you're right! Isn't this a legal problem or something? Commented Jun 25 at 19:52
  • 1
    Apparently not, nothing's changed so far ¯\_(ツ)_/¯ Commented yesterday
  • SO co doesnt read meta, they just post to it. Commented 7 hours ago
21

A minor issue but some questions have this weird spurious horizontal scroll. Is there a way to remove it, maybe fix the table format or something?

An example:

example page

1
  • 2
    Ah...!, yes, I mentioned it also in the last question: "- Yeah, I didn't understand why one question needed some agreement about sharing the answer publicly... - The question about "Which country you live in?" has the options not searchable, and the countries are not alphabetically sorted, it took me several minutes to find "Netherlands"... - Too many questions about AI... - The unneeded horizontal scrolling bar on many questions is annoying and not needed, as the Viewport is not even using 50% of my screen..." Commented Jun 23 at 17:02
21

Please don't assume everyone who's a programmer or uses Stack Overflow prefers "dark mode". And even for those who do, their preferences may change depending on their environment.

Personally, I can't take this survey because I can't read it.

Aside from my eyes not being as young as they used to be, my shiny computer screen becomes indistinguishable from a mirror in "dark mode". This is especially the case with the bright June solstice sun streaming in through my office Windows.

On most (all?) modern systems, you can read the user's preferred light/dark appearance mode in CSS. Please do that instead of making assumptions about dark mode.

3
  • Someone should make a survey whether people prefer White or Dark mode on Stack Overflow Commented Jun 25 at 6:58
  • 4
    there is a separate question for that meta.stackoverflow.com/q/439976/4117728 Commented Jun 25 at 8:26
  • And here I was thinking it finally actually looked at the user preference... Commented Jun 25 at 11:14
21

Stop asking about AI. Ask once if I use AI. If I say "no", ask me why, and I'll happily explain at length. And then stop asking questions about AI.

I said I didn't use AI, and then got three pages of questions asking about my use of dozens of AI tools.

An especially egregious question:

How important is source attribution when deciding whether to trust an AI-generated technical answer?

No option for "I don't, because it's AI." At least the rest mostly had an option for "no".

This is particularly relevant considering the amount of negative sentiment towards AI. (Even last year's survey reflects that. Other places do as well. Among many other places, see https://poll.qu.edu/poll-release?releaseid=3955 .)

New contributor
Josh Triplett is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
2
  • 2
    Survey methodology needs to be careful to not be self-reinforcing (beyond what selection bias already produces). If, for instance, this year's survey says that people trust AI more than last year, is that because AI has changed, because people's opinions have changed, or because more people bounced off this year because of the deluge of irrelevant questions? The survey already has logic to omit questions about things you said you don't work with, but it conspicuously doesn't have that with AI, and also asks loaded questions about AI like the one I quoted above. Commented yesterday
  • Also, no, it's not the same thing. People (hopefully) understand that a question asking about an IDE can also potentially apply to any editor; the tool is there, it's a difference in terminology. (Though the methodology would be improved by saying that explicitly so that people answer consistently.) That's not the same as asking a massive number of questions where the N/A answers are already implied by previous answers (and where those N/A answers are sometimes not present at all). Commented yesterday
19

What does "AI programming" mean?

One question asks (emphasis added):

In the last year, did you spend time learning AI programming or AI-enabled tooling on your own or at work?

It's not clear whether this refers to:

  • programming anything, using AI-powered tools (Claude Code, ChatGPT Codex, etc.)
  • programming, by any means, systems that employ AI

The next question then asked "How did you learn to code for AI in the past year?", which suggests the latter, but it's still ambiguous—"code for AI" is awkward, unclear phrasing.

17

I got pretty bored filling the survey in this year, I don't normally so maybe it's just not asking about the areas I'm interested in.

2
  • 18
    Let me guess: You're not intensely interested in AI tools? Commented Jun 24 at 18:53
  • 6
    @LyndonGingerich: And I selected that option. Do you use AI tools? No. Which of these AI websites did you use? None. Which of these AI plugins did you use? ... Which... Commented Jun 25 at 7:39
14

Below are some thoughts as I take the survey.

The first question is something that I've had a problem with for years, and I mentioned in my feedback. It doesn't do a good job of differentiating between what people do at work and what people do outside of work. At work, "I manage or support people who create code". But outside of work, "I create code occasionally". I'm not sure how to answer this, so I'm going to answer it related to my job. But as software developers, we're in a position to use our knowledge and skills in various ways, both at work and outside of work. This includes creating code, supporting people who create code, or doing something else with software (like writing documentation or testing) and this question continues to fail to capture those nuances.

The ordering of choices in the "how did you learn to code for AI in the past year" is quite strange. "Other online resources" comes before online resources like "blogs or podcasts", "online courses or certification", and "Stack Overflow or Stack Exchange", as one example. There's probably a better ordering, but any "other" should come after the choices it can be an other case of.

I can see that my feedback on the developer type role wasn't pulled in, and I have no idea how to answer that question. My title is "QA Manager", but "Developer, QA and test" doesn't seem like it matches. I'm not really an engineering manager or project manager, either, although I do use some of those skills. I went with "Other" in the end. But I can see other people struggling, too, such as blurry lines between "DevOps engineer" (which isn't a real thing, by the way, regardless of how many companies try to make it a thing) and "site reliability engineer" and "cloud infrastructure engineer" or "AI/ML engineer" and "Developer, AI apps or physical AI". It would be worth it to try to cut through job titles and get to a good list of types of work that people in and supporting development do, even if little definitions are needed so people can pick the right one.

The "which of the following activities are part of your work" is a better question than the developer role name question, but there's also confusion here. What's the difference between "writing, editing, or generating code" and "changing production code, systems, or infrastructure" and "deploying or releasing software"? The lines between those seem very blurry. Similarly, the lines between "debugging or troubleshooting", "monitoring systems logs, metrics, or alerts", "and investigating an incident, outage, or production issue" seem very close to each other. There are plenty of standards for types of work, tasks, and activities out there that can give a much better list with better boundaries.

There were a lot of questions on AI. I get that it's a hot issue and there are company reasons for also wanting these answers. However, in order to better track trends across the industry, having a lot more stability in questions year-over-year can help. Having more questions that are stable in the long-term would be more useful when the raw data is published later on. I'm not exactly sure what that looks like, but there are probably some good ideas in the feedback threads.

1
  • "The ordering of choices..." - no, I'd say this is not a bug - the order is random. Answer options are even shuffled every time you navigate to a next/previous question, and random order is usually good for survey hygiene. Only a few questions (like company size or role) are static. (although pinning "none of the above" and "other" to the bottom would be nice, it might not be supported by the platform) Commented Jun 26 at 10:18
11

This question about what I usually do when receiving an AI-generated answer for a development task is somewhat unclear and difficult to answer in a way that's not misleading.

When you receive an AI-generated answer for a development task, what do you usually do before using it? Select all that apply.

  • Run it locally
  • Read official docs
  • Search Stack Overflow
  • Ask a coworker
  • Inspect tests/security implications
  • Compare against codebase context
  • Use as-is
  • I do not use AI tools.

First, it's unclear what, exactly, it means to "receive an AI-generated answer for a development task". Does this include a coding agent writing code, or only answering a question?

Assuming it includes agents writing code, the real answer is "it depends on the task and the answer." The thing that I would most reliably do, "review the output and how the agent verified its work" is missing. "Inspect tests" is probably also usually applicable. Beyond that, it's situational:

  • If the code is incorrect, I would request changes to fix the issues I see. (This option is also missing)
  • If the agent didn't run the code, I would either run it locally myself or ask the agent to.
  • If the code calls an API I'm unfamiliar with, I might read official docs or search Stack Overflow.
  • I'm a freelancer, so I probably wouldn't ask a coworker.
  • If the code has security implications, I'd consider those, but most code doesn't.
  • If the code seems to be following a bad pattern, I may compare it against the codebase context to see if it's matching an existing bad pattern (which should be fixed) or failing to follow a good pattern (which should be emulated).
  • If, after considering all of the above, as well as applying my general professional judgement, it seems to be both correct and adequately tested, I'd use it as-is.
11

One problem that probably invalidates the entire survey was that some questions asked explicitly about working as a software developer, while others asked things like how many years of professional experience I have without specifying an industry or field. All the questions about my roles and responsibilities related to procurement and decision making seemed to assume I'm currently working as a software developer. I answered those questions as written. I work in a machine shop.

11

I perservered through "thinking about technologies you enjoy using for work, school, or personal projects, what is the main reason you would recommend one to others" question with its stupid grouping of work and personal tools into one bucket - at least it had "other" field - but gave up at "in a typical month over the past year, about how many distinct software tools, platforms, or services did you personally use to do your primary work". Who counts that???

10

It was buried in another answer, but let me highlight it here -- many of the questions were redundant, or could be fully derived from looking back at other questions.

I understand that this is a common survey tactic -- to ask the same question in a different way. It not only ensures accuracy, but it can also help prevent misinterpretation, as the user may not have understood the original question correctly.

Regardless, for how long this survey was, maybe it's better to be more direct with questions. The survey itself wasn't too long imo, but it felt long because of how many times I answered the same question. If it was all unique/novel questions, I would not have minded the length at all.

10

Here was the running list of concerns I ended up having while filling out the survey.

Links lead to the relevant question inside the survey's GitHub repo for better tracking and less screenshot clutter here.

The answers in the tech endorsement question are purposely shuffled, but the "I would not recommend any" and the "I'm not sure" options feel very weird being jumbled in with the rest of the "normal" answers, rather than down at the bottom alongside "Other: ___".

The question about job ease suffers from the same thing.

The question about job ease makes you select a single answer, but then the follow-up question is phrased ("For each factor that") and structured as though there should be multiple:

Job ease AI question

This isn't really a problem necessarily, but it reads awkwardly.

The immediately following pair of questions about what made your job harder suffers from the same thing.

The organization-level AI tool approach question felt poorly conceived; it asks you to select the "best description", but then provides multiple choices that aren't mutually exclusive.

My organization is building internal tools and different teams are choosing their own workflows; both are equally intrinsic to my organization's approach, but the question doesn't let me express that.

Overall, it just felt really long this year; the "18-20" minute estimate from the survey intro was way off base for me. This probably took me closer to 30 or 40 minutes, definitely not 20.

9

This question has no clear "No" answer, nor is it specified that you can skip answering this. enter image description here

2
  • 7
    Isn’t it clear that you can skip any question on any form anywhere on the web if it doesn’t have an asterisk next to it? Commented Jun 24 at 7:40
  • 3
    I never noticed that the mandatory questions even had an asterisk. Looking back at it, the signal-to-noise ratio is pretty bad. Commented Jun 24 at 8:07
9

For some reason, mend.io is actually formatted as a hyperlink in this question.

enter image description here

3
  • 4
    Presumably because it matches a regex that makes their system generate a link automatically. io is a valid TLD. Commented Jun 24 at 0:58
  • 2
    It's actually a full markdown link in the survey data on Github: github.com/StackExchange/Survey/blob/… Commented Jun 24 at 10:54
  • 9
    Sound slike that markdown was AI-generated as well... Commented Jun 24 at 11:40
8

This question should've had radio buttons for each row. It makes no sense that we can select all items. enter image description here

3
  • 2
    Meh; I'd be in favor of more validation to prevent nonsensical responses, but it appears intentional to me to allow you to select "I currently use AI for this" and also "I expect to use <more/ less> in the future" (which I did on multiple rows). Commented Jun 23 at 21:07
  • 4
    Expecting to use it "More" implies you're already using it to some extent, though. The wording is somewhat ambiguous. Commented Jun 23 at 21:15
  • I thought it was supposed to be a scale from 1 to 4 with an "N/A" answer on the far right. Commented Jun 24 at 18:54
4

Was I supposed to choose "None of the above" if I have a full-time job, but not (yet) retiring?

enter image description here

1

Are you meant to be able to take the survey multiple times, because I thought in previous years you could only take it once?

2
  • 7
    You want to take it more than once? I could not get through it even once so I tip my hat to you. Commented Jun 25 at 20:40
  • well there was one series response, and then there were several just click on anything responses Commented Jun 26 at 7:17

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.