Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2920 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.7131. Min Act: -0.0000

Data Index: 741287 (C4 (Web Text))

Max Activating Token Index: 958

Text #1

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.6401. Min Act: -0.0000

Data Index: 39780 (C4 (Web Text))

Max Activating Token Index: 150

Text #2

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.4625. Min Act: -0.0000

Data Index: 740108 (C4 (Web Text))

Max Activating Token Index: 1001

Text #3

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.5735. Min Act: -0.0000

Data Index: 184504 (C4 (Web Text))

Max Activating Token Index: 83

Text #4

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.6428. Min Act: -0.0000

Data Index: 1186834 (C4 (Web Text))

Max Activating Token Index: 304

Text #5

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.5168. Min Act: -0.0000

Data Index: 268467 (C4 (Web Text))

Max Activating Token Index: 388

Text #6

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.2626. Min Act: -0.0000

Data Index: 152329 (C4 (Web Text))

Max Activating Token Index: 379

Text #7

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.3436. Min Act: -0.0000

Data Index: 722918 (C4 (Web Text))

Max Activating Token Index: 227

Text #8

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.3206. Min Act: -0.0000

Data Index: 1048449 (C4 (Web Text))

Max Activating Token Index: 917

Text #9

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.4230. Min Act: -0.0000

Data Index: 148501 (C4 (Web Text))

Max Activating Token Index: 827

Text #10

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.5981. Min Act: -0.0000

Data Index: 864657 (C4 (Web Text))

Max Activating Token Index: 498

Text #11

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.2909. Min Act: -0.0000

Data Index: 1343552 (C4 (Web Text))

Max Activating Token Index: 139

Text #12

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.0537. Min Act: -0.0000

Data Index: 766097 (C4 (Web Text))

Max Activating Token Index: 454

Text #13

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.2479. Min Act: -0.0000

Data Index: 449748 (C4 (Web Text))

Max Activating Token Index: 391

Text #14

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.2584. Min Act: -0.0000

Data Index: 1196200 (C4 (Web Text))

Max Activating Token Index: 417

Text #15

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.0989. Min Act: -0.0000

Data Index: 723533 (C4 (Web Text))

Max Activating Token Index: 729

Text #16

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.2703. Min Act: -0.0000

Data Index: 611788 (C4 (Web Text))

Max Activating Token Index: 912

Text #17

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.2410. Min Act: -0.0000

Data Index: 149552 (C4 (Web Text))

Max Activating Token Index: 423

Text #18

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.4873. Min Act: -0.0000

Data Index: 936086 (C4 (Web Text))

Max Activating Token Index: 790

Text #19

Max Range: 2.7131. Min Range: -2.7131

Max Act: 2.2392. Min Act: -0.0000

Data Index: 695495 (C4 (Web Text))

Max Activating Token Index: 813