Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3502 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.6939. Min Act: -0.0000

Data Index: 1306112 (C4 (Web Text))

Max Activating Token Index: 767

Text #1

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.2619. Min Act: -0.0000

Data Index: 991191 (C4 (Web Text))

Max Activating Token Index: 904

Text #2

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.3258. Min Act: -0.0000

Data Index: 1046716 (C4 (Web Text))

Max Activating Token Index: 48

Text #3

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.1988. Min Act: -0.0000

Data Index: 659603 (C4 (Web Text))

Max Activating Token Index: 777

Text #4

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.2221. Min Act: -0.0000

Data Index: 967481 (C4 (Web Text))

Max Activating Token Index: 38

Text #5

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.1797. Min Act: -0.0000

Data Index: 439465 (C4 (Web Text))

Max Activating Token Index: 486

Text #6

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.2090. Min Act: -0.0000

Data Index: 1029307 (C4 (Web Text))

Max Activating Token Index: 584

Text #7

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.1764. Min Act: -0.0000

Data Index: 678502 (C4 (Web Text))

Max Activating Token Index: 436

Text #8

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.1542. Min Act: -0.0000

Data Index: 83280 (C4 (Web Text))

Max Activating Token Index: 40

Text #9

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.1188. Min Act: -0.0000

Data Index: 1177256 (C4 (Web Text))

Max Activating Token Index: 688

Text #10

Max Range: 1.6939. Min Range: -1.6939

Max Act: 0.9329. Min Act: -0.0000

Data Index: 507437 (C4 (Web Text))

Max Activating Token Index: 263

Text #11

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.0583. Min Act: -0.0000

Data Index: 1078950 (C4 (Web Text))

Max Activating Token Index: 397

Text #12

Max Range: 1.6939. Min Range: -1.6939

Max Act: 0.9895. Min Act: -0.0000

Data Index: 1008918 (C4 (Web Text))

Max Activating Token Index: 552

Text #13

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.0181. Min Act: -0.0000

Data Index: 749677 (C4 (Web Text))

Max Activating Token Index: 914

Text #14

Max Range: 1.6939. Min Range: -1.6939

Max Act: 0.9957. Min Act: -0.0000

Data Index: 1143501 (C4 (Web Text))

Max Activating Token Index: 801

Text #15

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.0967. Min Act: -0.0000

Data Index: 1160396 (C4 (Web Text))

Max Activating Token Index: 992

Text #16

Max Range: 1.6939. Min Range: -1.6939

Max Act: 0.9749. Min Act: -0.0000

Data Index: 1270369 (C4 (Web Text))

Max Activating Token Index: 723

Text #17

Max Range: 1.6939. Min Range: -1.6939

Max Act: 1.0416. Min Act: -0.0000

Data Index: 1006021 (C4 (Web Text))

Max Activating Token Index: 389

Text #18

Max Range: 1.6939. Min Range: -1.6939

Max Act: 0.9796. Min Act: -0.0000

Data Index: 457399 (C4 (Web Text))

Max Activating Token Index: 855

Text #19

Max Range: 1.6939. Min Range: -1.6939

Max Act: 0.9307. Min Act: -0.0000

Data Index: 745673 (C4 (Web Text))

Max Activating Token Index: 440