Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2264 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2594. Min Act: -0.0001

Data Index: 505225 (C4 (Web Text))

Max Activating Token Index: 1023

Text #1

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2581. Min Act: -0.0001

Data Index: 133738 (C4 (Web Text))

Max Activating Token Index: 91

Text #2

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2467. Min Act: -0.0001

Data Index: 647462 (C4 (Web Text))

Max Activating Token Index: 235

Text #3

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2497. Min Act: -0.0001

Data Index: 561817 (C4 (Web Text))

Max Activating Token Index: 98

Text #4

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2452. Min Act: -0.0001

Data Index: 56866 (C4 (Web Text))

Max Activating Token Index: 81

Text #5

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2415. Min Act: -0.0001

Data Index: 493357 (C4 (Web Text))

Max Activating Token Index: 545

Text #6

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2360. Min Act: -0.0001

Data Index: 1399723 (Python Code)

Max Activating Token Index: 968

Text #7

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2362. Min Act: -0.0001

Data Index: 162800 (C4 (Web Text))

Max Activating Token Index: 608

Text #8

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2343. Min Act: -0.0001

Data Index: 1104387 (C4 (Web Text))

Max Activating Token Index: 511

Text #9

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2356. Min Act: -0.0001

Data Index: 678 (C4 (Web Text))

Max Activating Token Index: 967

Text #10

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2406. Min Act: -0.0001

Data Index: 930246 (C4 (Web Text))

Max Activating Token Index: 191

Text #11

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2343. Min Act: -0.0002

Data Index: 567239 (C4 (Web Text))

Max Activating Token Index: 200

Text #12

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2327. Min Act: -0.0001

Data Index: 62117 (C4 (Web Text))

Max Activating Token Index: 769

Text #13

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2315. Min Act: -0.0001

Data Index: 472847 (C4 (Web Text))

Max Activating Token Index: 301

Text #14

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2397. Min Act: -0.0001

Data Index: 1124955 (C4 (Web Text))

Max Activating Token Index: 30

Text #15

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2316. Min Act: -0.0001

Data Index: 609352 (C4 (Web Text))

Max Activating Token Index: 218

Text #16

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2327. Min Act: -0.0001

Data Index: 484988 (C4 (Web Text))

Max Activating Token Index: 1023

Text #17

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2292. Min Act: -0.0001

Data Index: 1145658 (C4 (Web Text))

Max Activating Token Index: 502

Text #18

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2305. Min Act: -0.0001

Data Index: 1281054 (C4 (Web Text))

Max Activating Token Index: 987

Text #19

Max Range: 0.2594. Min Range: -0.2594

Max Act: 0.2365. Min Act: -0.0001

Data Index: 697130 (C4 (Web Text))

Max Activating Token Index: 68