Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3305 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 11.3878. Min Range: -11.3878

Max Act: 11.3878. Min Act: -0.0001

Data Index: 1522240 (Python Code)

Max Activating Token Index: 885

Text #1

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.9447. Min Act: -0.0000

Data Index: 1388296 (Python Code)

Max Activating Token Index: 619

Text #2

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.8439. Min Act: -0.0000

Data Index: 1386422 (Python Code)

Max Activating Token Index: 311

Text #3

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.8385. Min Act: -0.0000

Data Index: 1363488 (Python Code)

Max Activating Token Index: 177

Text #4

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.5112. Min Act: -0.0001

Data Index: 1438069 (Python Code)

Max Activating Token Index: 261

Text #5

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.6168. Min Act: -0.0001

Data Index: 1621111 (Python Code)

Max Activating Token Index: 168

Text #6

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.5044. Min Act: -0.0000

Data Index: 1491156 (Python Code)

Max Activating Token Index: 322

Text #7

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.4172. Min Act: -0.0001

Data Index: 1392692 (Python Code)

Max Activating Token Index: 268

Text #8

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.4105. Min Act: -0.0000

Data Index: 1435359 (Python Code)

Max Activating Token Index: 287

Text #9

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.5136. Min Act: -0.0001

Data Index: 1415486 (Python Code)

Max Activating Token Index: 158

Text #10

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.4945. Min Act: -0.0000

Data Index: 1594130 (Python Code)

Max Activating Token Index: 768

Text #11

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.3807. Min Act: -0.0000

Data Index: 1364388 (Python Code)

Max Activating Token Index: 265

Text #12

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.3776. Min Act: -0.0000

Data Index: 1522510 (Python Code)

Max Activating Token Index: 237

Text #13

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.3669. Min Act: -0.0000

Data Index: 1488371 (Python Code)

Max Activating Token Index: 649

Text #14

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.4939. Min Act: -0.0000

Data Index: 1486644 (Python Code)

Max Activating Token Index: 386

Text #15

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.3588. Min Act: -0.0000

Data Index: 1567240 (Python Code)

Max Activating Token Index: 318

Text #16

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.4803. Min Act: -0.0000

Data Index: 1451180 (Python Code)

Max Activating Token Index: 616

Text #17

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.3496. Min Act: -0.0001

Data Index: 1648651 (Python Code)

Max Activating Token Index: 824

Text #18

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.5839. Min Act: -0.0000

Data Index: 1453965 (Python Code)

Max Activating Token Index: 71

Text #19

Max Range: 11.3878. Min Range: -11.3878

Max Act: 10.2792. Min Act: -0.0000

Data Index: 1401806 (Python Code)

Max Activating Token Index: 119