Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2729 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.3138. Min Range: -1.3138

Max Act: 1.3138. Min Act: -0.0001

Data Index: 785559 (C4 (Web Text))

Max Activating Token Index: 71

Text #1

Max Range: 1.3138. Min Range: -1.3138

Max Act: 1.0478. Min Act: -0.0001

Data Index: 551360 (C4 (Web Text))

Max Activating Token Index: 835

Text #2

Max Range: 1.3138. Min Range: -1.3138

Max Act: 1.0904. Min Act: -0.0001

Data Index: 1172252 (C4 (Web Text))

Max Activating Token Index: 383

Text #3

Max Range: 1.3138. Min Range: -1.3138

Max Act: 1.0490. Min Act: -0.0001

Data Index: 871720 (C4 (Web Text))

Max Activating Token Index: 506

Text #4

Max Range: 1.3138. Min Range: -1.3138

Max Act: 1.0813. Min Act: -0.0001

Data Index: 284722 (C4 (Web Text))

Max Activating Token Index: 359

Text #5

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9593. Min Act: -0.0001

Data Index: 403837 (C4 (Web Text))

Max Activating Token Index: 617

Text #6

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9557. Min Act: -0.0001

Data Index: 910388 (C4 (Web Text))

Max Activating Token Index: 917

Text #7

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9856. Min Act: -0.0001

Data Index: 1285362 (C4 (Web Text))

Max Activating Token Index: 18

Text #8

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9277. Min Act: -0.0001

Data Index: 1029308 (C4 (Web Text))

Max Activating Token Index: 709

Text #9

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9405. Min Act: -0.0001

Data Index: 165818 (C4 (Web Text))

Max Activating Token Index: 260

Text #10

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9455. Min Act: -0.0001

Data Index: 1254143 (C4 (Web Text))

Max Activating Token Index: 68

Text #11

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9726. Min Act: -0.0001

Data Index: 606589 (C4 (Web Text))

Max Activating Token Index: 552

Text #12

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.8392. Min Act: -0.0001

Data Index: 472319 (C4 (Web Text))

Max Activating Token Index: 645

Text #13

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9439. Min Act: -0.0001

Data Index: 1088935 (C4 (Web Text))

Max Activating Token Index: 874

Text #14

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.8595. Min Act: -0.0001

Data Index: 296478 (C4 (Web Text))

Max Activating Token Index: 206

Text #15

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9263. Min Act: -0.0001

Data Index: 201537 (C4 (Web Text))

Max Activating Token Index: 1004

Text #16

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.8533. Min Act: -0.0001

Data Index: 1620655 (Python Code)

Max Activating Token Index: 40

Text #17

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.8232. Min Act: -0.0001

Data Index: 430480 (C4 (Web Text))

Max Activating Token Index: 54

Text #18

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.9587. Min Act: -0.0001

Data Index: 411723 (C4 (Web Text))

Max Activating Token Index: 631

Text #19

Max Range: 1.3138. Min Range: -1.3138

Max Act: 0.8411. Min Act: -0.0001

Data Index: 1045459 (C4 (Web Text))

Max Activating Token Index: 24