Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4529 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.4963. Min Range: -2.4963

Max Act: 2.4963. Min Act: -0.0000

Data Index: 1655480 (Python Code)

Max Activating Token Index: 581

Text #1

Max Range: 2.4963. Min Range: -2.4963

Max Act: 2.2699. Min Act: -0.0000

Data Index: 1501457 (Python Code)

Max Activating Token Index: 361

Text #2

Max Range: 2.4963. Min Range: -2.4963

Max Act: 2.2551. Min Act: -0.0000

Data Index: 1557929 (Python Code)

Max Activating Token Index: 381

Text #3

Max Range: 2.4963. Min Range: -2.4963

Max Act: 2.4094. Min Act: -0.0000

Data Index: 1538478 (Python Code)

Max Activating Token Index: 529

Text #4

Max Range: 2.4963. Min Range: -2.4963

Max Act: 2.1992. Min Act: -0.0000

Data Index: 1389440 (Python Code)

Max Activating Token Index: 412

Text #5

Max Range: 2.4963. Min Range: -2.4963

Max Act: 2.3212. Min Act: -0.0000

Data Index: 1579275 (Python Code)

Max Activating Token Index: 700

Text #6

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.9857. Min Act: -0.0000

Data Index: 1648488 (Python Code)

Max Activating Token Index: 483

Text #7

Max Range: 2.4963. Min Range: -2.4963

Max Act: 2.1698. Min Act: -0.0000

Data Index: 1560021 (Python Code)

Max Activating Token Index: 810

Text #8

Max Range: 2.4963. Min Range: -2.4963

Max Act: 2.4150. Min Act: -0.0000

Data Index: 1650675 (Python Code)

Max Activating Token Index: 437

Text #9

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.8078. Min Act: -0.0000

Data Index: 1533866 (Python Code)

Max Activating Token Index: 271

Text #10

Max Range: 2.4963. Min Range: -2.4963

Max Act: 2.1387. Min Act: -0.0000

Data Index: 1451597 (Python Code)

Max Activating Token Index: 587

Text #11

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.6919. Min Act: -0.0000

Data Index: 1478197 (Python Code)

Max Activating Token Index: 361

Text #12

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.6775. Min Act: -0.0000

Data Index: 1444200 (Python Code)

Max Activating Token Index: 832

Text #13

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.7034. Min Act: -0.0000

Data Index: 1465257 (Python Code)

Max Activating Token Index: 737

Text #14

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.8803. Min Act: -0.0000

Data Index: 1365555 (Python Code)

Max Activating Token Index: 15

Text #15

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.7083. Min Act: -0.0000

Data Index: 1654393 (Python Code)

Max Activating Token Index: 844

Text #16

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.7236. Min Act: -0.0000

Data Index: 1643318 (Python Code)

Max Activating Token Index: 962

Text #17

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.6100. Min Act: -0.0000

Data Index: 1581872 (Python Code)

Max Activating Token Index: 302

Text #18

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.9105. Min Act: -0.0000

Data Index: 1428032 (Python Code)

Max Activating Token Index: 410

Text #19

Max Range: 2.4963. Min Range: -2.4963

Max Act: 1.7253. Min Act: -0.0000

Data Index: 1469317 (Python Code)

Max Activating Token Index: 405