Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3935 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4270. Min Range: -1.4270

Max Act: 1.4270. Min Act: -0.0001

Data Index: 1318225 (C4 (Web Text))

Max Activating Token Index: 101

Text #1

Max Range: 1.4270. Min Range: -1.4270

Max Act: 1.2238. Min Act: -0.0001

Data Index: 1205393 (C4 (Web Text))

Max Activating Token Index: 459

Text #2

Max Range: 1.4270. Min Range: -1.4270

Max Act: 1.0477. Min Act: -0.0001

Data Index: 1167044 (C4 (Web Text))

Max Activating Token Index: 384

Text #3

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.8736. Min Act: -0.0001

Data Index: 984191 (C4 (Web Text))

Max Activating Token Index: 180

Text #4

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.9915. Min Act: -0.0001

Data Index: 883535 (C4 (Web Text))

Max Activating Token Index: 402

Text #5

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.9796. Min Act: -0.0001

Data Index: 801447 (C4 (Web Text))

Max Activating Token Index: 548

Text #6

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.8607. Min Act: -0.0001

Data Index: 1469069 (Python Code)

Max Activating Token Index: 730

Text #7

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.8604. Min Act: -0.0001

Data Index: 854664 (C4 (Web Text))

Max Activating Token Index: 249

Text #8

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.9052. Min Act: -0.0001

Data Index: 446408 (C4 (Web Text))

Max Activating Token Index: 982

Text #9

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.9555. Min Act: -0.0001

Data Index: 1195443 (C4 (Web Text))

Max Activating Token Index: 222

Text #10

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.8855. Min Act: -0.0001

Data Index: 727034 (C4 (Web Text))

Max Activating Token Index: 604

Text #11

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.8541. Min Act: -0.0001

Data Index: 671486 (C4 (Web Text))

Max Activating Token Index: 342

Text #12

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.8432. Min Act: -0.0001

Data Index: 638443 (C4 (Web Text))

Max Activating Token Index: 689

Text #13

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.8351. Min Act: -0.0001

Data Index: 727139 (C4 (Web Text))

Max Activating Token Index: 333

Text #14

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.8278. Min Act: -0.0001

Data Index: 634886 (C4 (Web Text))

Max Activating Token Index: 388

Text #15

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.8410. Min Act: -0.0001

Data Index: 1137103 (C4 (Web Text))

Max Activating Token Index: 249

Text #16

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.7463. Min Act: -0.0001

Data Index: 1091008 (C4 (Web Text))

Max Activating Token Index: 596

Text #17

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.7792. Min Act: -0.0001

Data Index: 481047 (C4 (Web Text))

Max Activating Token Index: 144

Text #18

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.7707. Min Act: -0.0001

Data Index: 260862 (C4 (Web Text))

Max Activating Token Index: 106

Text #19

Max Range: 1.4270. Min Range: -1.4270

Max Act: 0.7810. Min Act: -0.0001

Data Index: 1025767 (C4 (Web Text))

Max Activating Token Index: 420