Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3847 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6680. Min Act: -0.0002

Data Index: 509198 (C4 (Web Text))

Max Activating Token Index: 308

Text #1

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6657. Min Act: -0.0002

Data Index: 526300 (C4 (Web Text))

Max Activating Token Index: 434

Text #2

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6350. Min Act: -0.0002

Data Index: 656486 (C4 (Web Text))

Max Activating Token Index: 754

Text #3

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6306. Min Act: -0.0002

Data Index: 1302131 (C4 (Web Text))

Max Activating Token Index: 235

Text #4

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6286. Min Act: -0.0002

Data Index: 219351 (C4 (Web Text))

Max Activating Token Index: 135

Text #5

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6337. Min Act: -0.0002

Data Index: 182566 (C4 (Web Text))

Max Activating Token Index: 689

Text #6

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6331. Min Act: -0.0002

Data Index: 341494 (C4 (Web Text))

Max Activating Token Index: 491

Text #7

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6220. Min Act: -0.0002

Data Index: 451394 (C4 (Web Text))

Max Activating Token Index: 293

Text #8

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6165. Min Act: -0.0002

Data Index: 451887 (C4 (Web Text))

Max Activating Token Index: 886

Text #9

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6181. Min Act: -0.0002

Data Index: 1226090 (C4 (Web Text))

Max Activating Token Index: 245

Text #10

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6116. Min Act: -0.0002

Data Index: 1078508 (C4 (Web Text))

Max Activating Token Index: 544

Text #11

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6057. Min Act: -0.0002

Data Index: 415616 (C4 (Web Text))

Max Activating Token Index: 577

Text #12

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.5914. Min Act: -0.0002

Data Index: 956073 (C4 (Web Text))

Max Activating Token Index: 368

Text #13

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6135. Min Act: -0.0002

Data Index: 594111 (C4 (Web Text))

Max Activating Token Index: 83

Text #14

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.5993. Min Act: -0.0002

Data Index: 219794 (C4 (Web Text))

Max Activating Token Index: 627

Text #15

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6026. Min Act: -0.0002

Data Index: 746346 (C4 (Web Text))

Max Activating Token Index: 447

Text #16

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.5974. Min Act: -0.0002

Data Index: 289143 (C4 (Web Text))

Max Activating Token Index: 678

Text #17

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.5810. Min Act: -0.0002

Data Index: 1126569 (C4 (Web Text))

Max Activating Token Index: 483

Text #18

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.6032. Min Act: -0.0002

Data Index: 966816 (C4 (Web Text))

Max Activating Token Index: 593

Text #19

Max Range: 0.6680. Min Range: -0.6680

Max Act: 0.5932. Min Act: -0.0002

Data Index: 1131649 (C4 (Web Text))

Max Activating Token Index: 851