Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2883 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1085. Min Range: -2.1085

Max Act: 2.0832. Min Act: -0.0000

Data Index: 531621 (C4 (Web Text))

Max Activating Token Index: 378

Text #1

Max Range: 2.1085. Min Range: -2.1085

Max Act: 2.0327. Min Act: -0.0000

Data Index: 812933 (C4 (Web Text))

Max Activating Token Index: 840

Text #2

Max Range: 2.1085. Min Range: -2.1085

Max Act: 2.1085. Min Act: -0.0000

Data Index: 1085200 (C4 (Web Text))

Max Activating Token Index: 723

Text #3

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.9331. Min Act: -0.0000

Data Index: 717395 (C4 (Web Text))

Max Activating Token Index: 1019

Text #4

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.8001. Min Act: -0.0000

Data Index: 155526 (C4 (Web Text))

Max Activating Token Index: 938

Text #5

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.7819. Min Act: -0.0000

Data Index: 726833 (C4 (Web Text))

Max Activating Token Index: 210

Text #6

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.6416. Min Act: -0.0000

Data Index: 95045 (C4 (Web Text))

Max Activating Token Index: 776

Text #7

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.6378. Min Act: -0.0000

Data Index: 704747 (C4 (Web Text))

Max Activating Token Index: 394

Text #8

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.6970. Min Act: -0.0000

Data Index: 1247178 (C4 (Web Text))

Max Activating Token Index: 845

Text #9

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.6911. Min Act: -0.0000

Data Index: 924662 (C4 (Web Text))

Max Activating Token Index: 401

Text #10

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.5533. Min Act: -0.0000

Data Index: 116711 (C4 (Web Text))

Max Activating Token Index: 409

Text #11

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.6353. Min Act: -0.0000

Data Index: 806901 (C4 (Web Text))

Max Activating Token Index: 20

Text #12

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.5852. Min Act: -0.0000

Data Index: 1121196 (C4 (Web Text))

Max Activating Token Index: 820

Text #13

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.4824. Min Act: -0.0000

Data Index: 60990 (C4 (Web Text))

Max Activating Token Index: 323

Text #14

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.5299. Min Act: -0.0000

Data Index: 569301 (C4 (Web Text))

Max Activating Token Index: 965

Text #15

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.4284. Min Act: -0.0000

Data Index: 232196 (C4 (Web Text))

Max Activating Token Index: 305

Text #16

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.5268. Min Act: -0.0000

Data Index: 536780 (C4 (Web Text))

Max Activating Token Index: 415

Text #17

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.6154. Min Act: -0.0000

Data Index: 612627 (C4 (Web Text))

Max Activating Token Index: 346

Text #18

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.5643. Min Act: -0.0000

Data Index: 308695 (C4 (Web Text))

Max Activating Token Index: 476

Text #19

Max Range: 2.1085. Min Range: -2.1085

Max Act: 1.5861. Min Act: -0.0000

Data Index: 552515 (C4 (Web Text))

Max Activating Token Index: 910